A Novel Method for Function Smoothness in Neural Networks

Existing methods for function smoothness in neural networks have limitations. These can make training sensitive to their hyperparameters, or constraints limit model capacity. impose too much smoothness, even areas without data, they non-meaningful constraints. The way these measure also be computationally hard. One of the main Lipschitz continuity, does not imply differentiability theoretically, let alone continuous differentiability, that is smoothness. In this paper, we propose a method based on theoretical definition derivative ensure parametrized should tend toward its value given network parameters vicinity samples. changes classifier and minimally has no added hyperparameters. proposed shown achieve smoother both testing samples all tested datasets, as measured with decreased values Frobenius norm Jacobian respect inputs. Due correlation between generalization, makes classifiers generalize better higher accuracy than default Restricted ImageNet, CIFAR10 MNIST. adversarial robustness, high-capacity architecture more robust generated PGD attack compared CIFAR10, Fashion-MNIST MNIST datasets.